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ILLINOIS IMPROVING TEACHER QUALITY STATE GRANTS: 
LEARNING ABOUT STEM PARTNERSHIPS 


As a necessary> preliminary for this issue on Illinois ’ experience 
with school-university partnerships to provide teacher development op¬ 
portunities, this article provides a review of the literature relevant to such 
programs. As the evaluation and assessment have changed over the years 
since the Eisenhower grants period, the author explains the thoughtful 
and purposive ways in which the assessment of Illinois’ programs have 
evolved. These changes touch the structures, processes, meta-evaluation, 
and capacity> for evaluation. Illinois Board of Higher Education grants 
policies now reflect the results of these evolved evaluation methods. The 
author includes details of research methods and protocols, research ques¬ 
tions, and the theory of change that informs all these methods. 


This article provides background and an overview for this spe¬ 
cial issue of Planning and Changing on school-university partnerships de¬ 
signed to support teacher professional development and learning in Science, 
Technology, Engineering, and Math (STEM) disciplines. The enterprise 
we describe is the Illinois Improving Teacher Quality (ITQ) state grants 
program administered by the Illinois Board of Higher Education (IBHE) 
and evaluated by the Center for the Study of Education Policy (CSEP) 
from 2003 into 2011. This special issue explores the relationships among 
partnership structures, collaborative partnership processes, program meta¬ 
evaluation and evaluation capacity building, and policy refinement. This 
article also offers a brief overview of the scholarship, a research frame¬ 
work, research questions, and methodology, the comparative case study. 
We share our research protocol (Appendices A-C: Illinois Teacher Qual¬ 
ity State Grants: Research Framework) and offer three articles (named be¬ 
low) as our comparative case findings to date on the partnership structur¬ 
al configurations; partnership collaborative implementation features and 
processes; and the IBHE’s theory of change and use of program theory. 
Finally, we follow these three articles from select ITQ projects as illustra¬ 
tive case studies considered through the lens of program theory, and we 
list these below as well. Overall, this article provides the balcony view of 
the project and evaluation development from 2004 to 2011. 

Background 

In 2001, the Elementary and Secondary Education Act of 1965 
(ESEA) was reauthorized and amended and after 2002 was known as the 
No Child Left Behind Act (NCLB). This act “places significant empha- 
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sis on assisting schools and school districts in increasing the academic 
achievement of all students by improving teacher and principal quality 
and ensuring that all teachers are highly qualified” (IBHE, 2003). In Title 
IIA, NCLB was reconceived as the Improving Teacher Quality (ITQ) state 
grant program. The ITQ program replaced and expanded the Dwight D. 
Eisenhower Professional Development Program, although most ITQ proj¬ 
ects retain Eisenhower’s STEM focus. Hence, this special issue addresses 
the STEM projects, alone from the total collection of projects for reasons 
expanded below. The policy also mandated partnership, defining that term 
by listing eligible institutional types that at minimum include a P-12 and 
a postsecondary partner. 

The NCLB Title IIA ITQ grants were in their inaugural year when 
we began supporting individual project evaluation and serving as meta¬ 
evaluators. Like most grant evaluators, we do not design the programs we 
evaluate nor do we dictate evaluation parameters on individual projects. 
Since 2004, we have become partners with IBHE grants administrators 
as they worked to make evaluation meaningful. Evaluation is an artifact 
of government activism. As funding grows, so does the demand for eval¬ 
uation with shareable, usable results (Office of Budget & Management, 
1993; Joint Committee on Standards for Educational Evaluation, 1994). 
The intention is to build and refine policy through successive iterations, 
use evaluation to study the theories of change represented in each grant 
project, and share findings across audiences: policy makers, scholars, and 
practitioners. The ITQ state grant program is one resource to study change 
mechanisms facilitated and sustained by collaborative partnerships. 

In 2003-2004, we re-conceptualized the IBHE grant funding process 
as a set of comparable “treatments” that could be evaluated and improved by 
linking evaluation to the grant funding cycle. The cycle begins anew with 
a new request for proposals (RFP) or renewal applications that reflect what 
evaluators have learned across the grants. Since 2007, we use program theory 
and its visual organizer, the logic model, to clarify theories of change, make 
them explicit, and consider theory development and refinement. 

Throughout the development and refinement of our approach to 
STEM school-university professional learning partnerships, we have re¬ 
lied on scholarship in several areas, chief among them school-university 
partnerships, professional development, and educational renewal or change 
processes. Our focus on change processes draws out the structures, imple¬ 
mentation features, and evaluation capacity building (ECB) processes nec¬ 
essary among partners to make these projects result in viable, sustainable 
partnerships to which teacher and student learning can be attributed over 
time. We briefly share our research framework here. We follow with three 
articles, each of which explores a key element of our work: the nature of 
partnerships in “Three Configurations of School-University Partnerships: 
An Exploratory Study” (Baker, 2011); the role of partnerships in renew¬ 
ing educational institutions in “Characteristic Collaborative Processes in 
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School-University Partnerships,” (Gardner, 2011); and building evaluation 
capacity among partners in “Evaluation Capacity Building in a School-Uni¬ 
versity Partnership Grant Program” (Haeffele, Hood, & Feldmann, 2011). 
Each of these articles offers a deeper exploration of scholarship and how it 
has guided us as we endeavor to answer four research questions. 

Research Questions 

There are four basic research questions that guide the statewide 
evaluation of the ITQ grant program. The goal is to define what partner¬ 
ship means for ITQ projects, individually and collectively. With empiri¬ 
cal characterizations of the partnerships in hand, we then develop expla¬ 
nations for forming and sustaining viable partnerships capable of yielding 
changes in student learning and achievement. These questions remain es¬ 
sentially unchanged since 2004: 

1) How are the ITQ STEM professional development partnerships struc¬ 
tured? 

2) How do collaborative implementation processes, past, present, and 
emerging, assure achievement of ITQ’s educational goals? 

3) How does collaborative evaluation guide decisions to keep the fo¬ 
cus on teacher learning, implementation, and student learning and 
achievement? 

4) How can IBHE and school-university partners build greater capacity 
through systemic program evaluation? 

We approach these questions using a comparative case approach to first 
describe, then develop explanations for, the effectiveness and sustain¬ 
ability of partnerships. We discuss our research frameworks, designs, and 
methods below. 


The ITQ Research Framework 

Three strands of scholarship inform our ITQ grant meta-evalua- 
tions from 2004 to the present: (a) school-university collaborative part¬ 
nerships in STEM disciplines; (b) partnership implementation features 
and processes for professional learning; and (c) evaluation and evalu¬ 
ation capacity building. In 2006, we developed our research protocol 
which we revised in 2010 (Appendices A-C: Illinois Teacher Quality 
State Grants: Research Frameworks). We begin with the structures of 
these partnerships (Appendix A: Structures: Collaborative Elements) and 
the collaborative implementation features and processes of the partner¬ 
ships (Appendix B: Professional Learning Processes: Collaborative Ele¬ 
ments). Finally, we consider evaluation capacity building using program 
theory and logic modeling (Appendix C: Evaluation Capacity Building: 
Collaborative Elements). 
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This overview is necessarily brief, and individual articles include 
the literature reviews of the topics at hand. Here, we offer an overview on 
the problems of understanding partnerships shared by scholars, evalua¬ 
tors, and practitioners. We do not review the scholarship on effective pro¬ 
fessional development in-depth here but will take it up in the later articles 
in this special Planning and Changing issue. There is a long standing con¬ 
sensus about professional development (Hawley &Valli, 1999). We focus 
on an empirical characterization of partnerships, so we consider the attri¬ 
butes of professional development in this light. In 2009, the IBHE adopted 
a policy for the ITQ grants requiring that all projects include five principal 
characteristics of effective professional development: (a) content focus; 
(b) active learning; (c) coherence; (d) duration; and (e) collective partici¬ 
pation (Desimone, 2009). Professional learning is strongest when actual 
classroom models, instructional tools, and materials are shared resources, 
and several ITQ projects incorporate this insight (Ball & Cohen, 1996). 
Because these characteristics are part of policy, they become foci for eval¬ 
uation and organize our thinking about professional development as a way 
to examine the partnership structures, collaborative implementation pro¬ 
cesses, and evaluation capacity. 

Understanding Partnerships: Structure and Implementation 

Partnerships are common in human experience. Marriage unites 
life partners. Professional partnerships unite doctors and lawyers; business 
partnerships unite entrepreneurs. Partnerships are at their best when part¬ 
ners exhibit a spirit of hope for what they can accomplish and share a will¬ 
ingness to work together through challenges. School-university partner¬ 
ships come with particular challenges, and stakes for the partners are high. 
Professional development funds are increasingly scarce. Partnerships take 
time and often rely on only a few individuals to make them viable and sus¬ 
tainable. School and university partners differ in their approaches to edu¬ 
cation and to reform, and wave after wave of reform has made all the part¬ 
ners anxious about what they can hope to accomplish and distrustful of 
each other (Bryk & Schneider, 2002; Gardner, 2010). 

In scholarly terms, school-university partnerships are not well-de¬ 
fined (Clifford & Millar, 2008; Zhang et al., 2009). Goodlad defines part¬ 
nerships between schools and universities as “a planned effort to establish 
a formal, mutually beneficial interinstitutional relationship (1991, p. 58),” 
and this definition is the most commonly cited (Clifford & Millar, 2008). 
Another common approach defines partnership by membership (Kingsley 
& Waschak, 2005; Podolny & Page, 1998). IBHE took this approach in its 
requests for proposals (RFPs) and lists required and potential partners. Re¬ 
quired higher education partners must include a public or private college or 
university, including two divisions: the unit that prepares educators (e.g., 
College of Education) and the unit that includes discipline specific depart- 
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ments that represent content expertise in STEM (e.g., College of Arts and 
Sciences). It is interesting to note that Illinois ITQ does not include any divi¬ 
sions of engineering, applied science, or technology, and the RFPs have not 
specified these partners on membership lists. An Illinois public school dis¬ 
trict meeting federal criteria as “high need” is another required partner. Op¬ 
tional partners include community colleges, regional offices of education, a 
wide range of community partners, and private schools (IBHE, 2010). This 
list of partners as members constitutes Illinois’ de facto partnership policy: 
bricks and mortar as partner to bricks and mortar. 

Other definitions and approaches to understanding partnerships 
rely on comparisons to other organizational and interinstitutional forms 
such as networks, consortia, and alliances (Badiali & Flora, 2000; Clifford 
& Millar, 2008; Druckman & Peterson, 2002; Podolny & Page, 1998). 
Few scholars acknowledge the problem of partner asymmetry developed 
in this special issue of Planning and Changing (Baker, 2011; Gardner, 
2011; Krasny, 2005). The NCLB Act favored an innovation-adoption 
model that leverages university expertise to alter school practices (Krasny, 
2005), so an expert-driven, top-down orientation to partnership that privi¬ 
leges the higher education partner is a design element in all ITQ projects 
to one degree or another. 

The Improving Teacher Quality research framework (Appendix 
A: Structures: Collaborative Elements) uses four lenses to take a nuts and 
bolts approach to describing the ITQ partnerships structurally with: 

1) General descriptions of the institutions and individuals that comprise 
the partnership, emphasizing the formal characterizations and the role 
of authority in the partnership; 

2) Characterizations of the roles and relationships that are crucial to 
partnership and how they are distributed, including the all-significant 
boundary-spanners crucial to partnership viability; 

3) Accounts of formal and informal rules and expectations among the 
partners; and 

4) Resource audits that include partnership expertise, financial resources, 
and in-kind contributions. 

Each of the four lenses represents an element of collaboration 
that is basic and shared by ITQ partnerships. Understanding partnership is 
problematic, and the struggles we share with other researchers and eval¬ 
uators testify to this. Further compounding our challenges to understand 
how partnerships can enhance student achievement is that they are not 
formed once and remain the same forever. They move forward in time and 
must develop and adapt. It is insufficient to understand partnerships as 
structures or complexes of roles, rules, expectations, and responsibilities. 
Therefore, we extend our work into the implementation features and pro¬ 
cesses that we observe empirically in our ITQ statewide evaluation. 
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The ITQ research framework (Appendix B: Professional Learning 
Processes: Collaborative Elements) considers collaborative implementa¬ 
tion features and processes by organizing partnership development as a 
three-stage chronology: 

1) The partnership history or background prior to proposing an ITQ grant 
and changes that ITQ generated in existing partnerships. Prior rela¬ 
tionship is one of the best predictors of a partnership with develop¬ 
mental potential; 

2) The partnership’s processes that occur when planning the project, 
making decisions, moving forward with plans, and taking steps to 
make the partnership sustainable; and 

3) The partnership’s plans and vision for the future as late steps that link 
the partnership’s characteristic structures and implementation pro¬ 
cesses to evaluation, monitoring, and improvement or ECB. 

We can also describe the ITQ research framework by what it does 
not do. It does not set out to study something as ineffable as trust, although 
we know that trust matters in partnerships (Bryk & Schneider, 2002). We 
understand the value of ineffable qualities like trust and mutual respect to 
any partnership, and indeed we observed these in different contexts and 
different degrees of development. But we opted for basic descriptions of 
structures and implementation practices and processes instead. In all part¬ 
nerships, structural and collaborative features enable evaluation and use of 
evaluation findings and implications to one degree or another, so our re¬ 
search delved into the partnerships to consider how to support evaluation 
for improvement and building evaluation capacity as critical partnership 
functions and as a statewide ECB mandate (Preskill & Boyle, 2008; Stock- 
dill, Baizerman, & Compton, 2001). 

Evaluation and Evaluation Capacity Building 

The ITQ research framework separates evaluation and evaluation 
capacity building in its statewide evaluation research into IBHE-funded 
partnerships. In the first instance, evaluation assesses the adequacy of ITQ 
project evaluation plans to inform all partnership constituencies about re¬ 
sults and to make ongoing adjustments using this feedback. Key collabor¬ 
ative elements in evaluation are the intended goals and outcomes and their 
appropriateness for ITQ, the adequacy of planned measures, and the ad¬ 
equacy of evaluation to inform improvement and to provide funders at the 
IBHE with tools they need to make funding decisions. Evaluation capac¬ 
ity building is a set of intentions and actions to make evaluation integral to 
organizational and interinstitutional learning by building understanding of 
evaluation, improving evaluation practices, and making the use of evalua¬ 
tion results a partnership imperative. Through ECB, partnership constitu¬ 
encies “learn to think evaluatively and how to engage in sound evaluation 
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practice” (Preskill & Boyle, 2008, p. 443). The ITQ research framework 
(Appendix C: Evaluation and Evaluation Capacity Building: Collabora¬ 
tive Elements) investigates this as a collaborative capacity of partnerships, 
considering three features: 

1) Development of an explicit theory of change that scholarship and 
practical wisdom would suggest comprise a reasonable plan to im¬ 
prove student achievement; 

2) Feedback mechanisms as structures and processes to enable partners 
to learn together to improve the partnership and the project; and 

3) Ability of project evaluations to support statewide capacity to improve 
professional development partnerships. 

Evaluation capacity building is a related set of collaborative ele¬ 
ments that the partners must share. Among these are shared responsibili¬ 
ties for developing a theory of change for the project and developing a log¬ 
ic model (Chen, 2005; Chen & Rossi, 1987; Frechtling, 2007; (Wholey, 
Elatry, & Newcomer, 2004). ITQ projects must present detailed evaluation 
plans that include logic models based on the scholarship on STEM profes¬ 
sional development, partnerships, and exemplary evaluation practices. In 
this special issue, a detailed discussion of ECB in ITQ offers an overview 
of the use of theories of change and logic modeling in ITQ ECB develop¬ 
ment efforts (Haeffele et al., 2011). One feature of ECB in Illinois is that 
individual project evaluations are just one piece of the state’s evaluation 
strategy. The second is the statewide evaluation using comparative case 
studies that has allowed the ITQ program to develop and refine STEM ed¬ 
ucation policy at IBHE. In the next section, we offer the CSEP statewide 
evaluation designs and methods. 

Comparative Case Studies in ITQ Meta-Evaluation 

We have written at length on single and comparative case stud¬ 
ies in research, evaluation, and the development of theory (Vogt, Gard¬ 
ner, Haeffele, & Baker, 2010). In the case of grant evaluations, compara¬ 
tive case studies develop as natural experiments that consider a full set of 
programs for evaluation (Stake, 2006; Vogt et al. 2010). The practice of 
studying STEM professional development efficacy using multiple, com¬ 
parative cases is long-standing in former Eisenhower grants and their an¬ 
tecedents in Illinois ITQ, National Science Foundation Math Science Part¬ 
nerships, and other grants (Abell et al., 2007; Blackwell, 2004; Birman, 
Reeve, & Sattler, 1998; Boyd, Banilower, Pasley, & Weiss, 2003; Garet, 
Porter, Desimone, Birman, & Yoon, 2001; National Network of Eisen¬ 
hower Regional Consortia & Clearinghouse, 2004a, 2004b; Porter, Garet, 
Desimone, Yoon, & Birman, 2000). 

In our comparative work, we have been guided by three goals: (a) 
to evaluate the state program to help improve it; (b) to provide technical 
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assistance to individual projects; and (c) to research what works in these 
partnerships in general. We work to balance our evaluation and research 
roles with our technical assistance to projects, and we do not make funding 
decisions. We began with a comparative orientation, realizing that com¬ 
parisons are empirically rooted in individual cases. But a case by case re¬ 
port is of little value to policy makers, so the IB HE sought our assistance 
to apply lessons learned from each grant cycle in subsequent cycles. We 
understood that we sacrificed some situational variation to develop expla¬ 
nations for policy application. We also understood the power of variation 
to help explain a phenomenon of interest. In the next section, we offer a 
general overview of comparative cases, the role of sampling and analytic 
purpose, and the role of variation in theory development. 

Comparative Cases: Sampling, Analytic Purpose, and Variation 

We began in 2004 to empirically describe partnerships. These proj¬ 
ects, designed under the same policy regime and scholarship, share com¬ 
monalities and allow us to see the same general program theory applied 
in different contexts. Our work continues as we seek explanations of what 
makes partnerships viable and sustainable. This requires multiple cases with 
sufficient variation for building, refining, and testing theories of change 
within and across the ITQ projects. We briefly discuss design, but we focus 
on the adequacy of our sample to match our analytic purposes and how vari¬ 
ation helps explain what works and what doesn’t (Vogt et al., 2010). 

One benefit of case studies is flexibility and the variety of appli¬ 
cable designs. In our meta-evaluation, we employ four data collection de¬ 
signs: interviews, archival analysis of grant-wide and project specific doc¬ 
uments and other materials, focus groups, and naturalistic and participant 
observations. We collect data through site visits, statewide symposia, web 
site analysis, and electronic file transfer. 

The ITQ meta-evaluation sample for data collection is the full 
project set from 2004, a total of 22 STEM projects out of 26 projects. 
Generally since 2004, the sample shra nk as the policy was re-focused, 
and fewer projects were funded. 2007 was a year where policy changed 
course, and since that time fewer projects are funded as a matter of course 
in order to focus on identifying the structures, processes, and evaluation 
capacity of effective projects. Ten cases were funded in FY2010 and nine 
in FY2011 as one project did not apply for continued funding. 

We have two sampling strategies for collecting individual project 
data. Both are purposive. The first is to complete two to five annual site visits 
to observe planning and steering meetings, summer institutes, school-based 
teams, follow-up workshops, and classroom visits. We return to sites based 
on research questions as we develop them for individual cases. The more 
complex and interesting the case and the more we believe we have to learn, 
the more frequent the visits. These decisions are made collaboratively by the 
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CSEP team. The second sampling strategy is convening project directors, ex¬ 
ternal evaluators, and staff and self-identified partners for annual or biannual 
symposia that include focus groups. We collect all public documents and ma¬ 
terials developed for project events, project record keeping and communica¬ 
tions, and school products that may include student work. Increasingly web- 
based resources are sampled, including video and MP3 links. 

In statewide program evaluations, typically comparative case sam¬ 
ples are intermediate sized and use the full set of participating cases. This 
suits our analytic purposes as well. For preliminary, descriptive work, a sin¬ 
gle case studied over time and in-depth is useful. But for explanatory pur¬ 
poses, intermediate samples are needed: 10 to 40 cases selected for varia¬ 
tion depending on the research questions (Ragin, 2008; Vogt et al., 2010). 
The use of discontinuing cases is well-established for sampling and analy¬ 
sis (Patton, 2002; Vogt, et al., 2010), and little existing partnership research 
includes such cases (Clifford & Millar, 2008). Comparative cases with dis- 
confirming cases, sensitivity to contextual variation (Abell et al. 2007), and 
pairing cases for analysis allowed the team to develop characterizations of 
STEM professional development partnerships as single-tier, multi-tier, or 
complex-brokered (Baker, 2011). We wanted to use a range of confirming 
and discontinuing cases to bolster our confidence that we had: (a) accurate¬ 
ly described partnerships as structures and processes and (b) preliminari¬ 
ly explained partnership viability and sustainability for policy makers and 
other constituents. Without explanations, policy makers would not be able 
to use evaluation results for funding and new policy iterations. 

We also benefitted from this intermediate sample size to study 
partnership variations. Illinois is a state of extremes. We would not expect 
a P-20 partnership to be the same for elementary teachers in Chicago as it 
might be for high school teachers in the rural south. ITQ projects represent 
a rich array of rural, mid-sized and large urban, and mixed settings, and all 
levels of schooling (i.e., elementary, middle, and high). The postsecond¬ 
ary partners vary by type, from regional and flagship public universities, 
private research universities, to small liberal arts colleges. 

These and other variations mattered as the ITQ grants face con- 
text-specific challenges. For example, we wanted to understand effective 
STEM professional learning structures in context. Collaborative models for 
professional learning are well-developed in scholarship, but creating these 
structural arrangements and effective team processes in different settings 
is challenging (McLaughlin & Talbert, 2006; Stoll & Louis, 2007). Most 
schools convene teams organized by grade level in elementary school, pods 
in middle school, and departments in high school. Project-specific respons¬ 
es to these challenges were critical to developing our three-part structural 
configurations model to shape policy and improve P-20 collaborations, of 
which we use three cases to highlight in this special issue. The availabil¬ 
ity of university partners in Chicago means that urban ITQ projects must 
compete with other initiatives from neighboring universities. Without 
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school leadership focused on STEM, teacher energies are easily drained 
away. The University of Chicago’s ITQ case study, “Seeking Symmetry in 
a School-University Partnership: University of Chicago and Chicago Pub¬ 
lic Schools—A Collaborative Approach to Developing Models and Tools 
for Professional Development and Teacher Preparation” (Leslie, 2011), 
presented in this special issue, had team-based structures and processes 
in its 2007 design, but struggled as some principals encouraged multiple 
initiatives, with a resulting loss of coherence, creating variable results for 
school-based teams (Newmann, Smith, Allensworth, & Bryk, 2001). In 
contrast, Southern Illinois University-Edwardsville is the regional partner 
to several remote, poorly resourced schools, sometimes with only one sci¬ 
ence teacher. For the Edwardsville ITQ case study, “Improving Science 
Instruction in Southwestern Illinois and Metro East St. Louis: Students 
Learning Science through a Sustained Network of Teachers,” also in this 
special issue, one challenge has been to create networked support teams 
and other resources for isolated teachers. 

In a final example, the case study of Southern Illinois Universi- 
ty-Carbondale’s project, “Improving Teacher Quality in Southern Illinois: 
Rural Access to Mathematics Professional Development (RAMPD)” (Pru- 
saczyk & Baker, 2011) included here, is a cautiously optimistic story about 
a partnership configuration where we would not expect success under our 
own current theories. Yet the RAMPD project is developing a regional net¬ 
work of elementary math teachers who are increasingly capable of meet¬ 
ing the challenges of radically changing teaching practices and leading the 
way for others. The variation that makes the case unique informs our un¬ 
derstanding of the other nine cases. It is an exception that demonstrates a 
developing rule. 

The interplay between ITQ cases, developed over years by the 
Center team of six site visitors/evaluators, leads us to adopt a comparative 
case study approach that has been applied to evaluating the professional 
development component of STEM partnerships. In the next section, we 
consider our coding and analysis procedures. 

Coding and Analysis 

In our approach to evaluation, we consciously break with the trou¬ 
blesome quantitative/qualitative divide to distinguish data that is best coded 
as words from that best coded as numbers or symbols. We choose instead 
to emphasize sampling, analytic purposes, the role of variation, and the ap¬ 
plication of findings to policy (Vogt, Gardner, & Haeffele, in press). We de¬ 
veloped several codes from our data set, but our major coding scheme uses 
words with documentary, interview, and observational evidence as major 
data sources. Numeric coding was used generally within projects for as¬ 
sessment results (including state tests) and project demographics. Numeric 
codes are most useful for individual projects in their own evaluations which 
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are shared in the annual renewal process. These numeric codes are found in 
the three case studies as needed and did not contribute substantively to our 
findings about structures and processes, or description of ITQ’s evaluation 
capacity building efforts, the three foci of this special issue. 

From our comparisons over time, we developed a truth table as a 
set of codes. We developed nine key variables that contributed to our under¬ 
standing of collaborative partnerships. We discuss the truth table at length 
elsewhere but offer this overview of the independent variables we consid¬ 
ered (Vogt et al., 2010). The dependent variables are teacher learning, imple¬ 
mentation that results from professional development, and student learning. 
Each variable figured in the shaping of ITQ policies at the IBHE from 2004 
on, albeit with different emphases as we learned together what mattered 
for collaborative partnerships. From this coding scheme, we developed two 
models of partnership structures and processes, the structures of training 
and implementation processes model (STPI) (Baker & Gardner, 2005) and 
“Three Configurations of School-University Partnerships” (Baker, 2011). 
“Characteristic Collaborative Processes in School-University Partnerships” 
(Gardner, 2011), the processes typical of each structural configuration are 
considered in this special Planning and Changing issue. 

Each ITQ Center researcher was responsible for data collection 
for each grant project, sometimes with a second researcher. Each visit was 
prepared as contemporaneous field and interview notes. Field notes were 
then developed into memos. These memos were shared among the Center 
team and IBHE grants administrators. Using the memos, the Center team 
held coding seminars four to eight times annually. Additionally, Center 
evaluators and IBHE administrators convened to review results once or 
twice a year, with all members reviewing memos and major project docu¬ 
ments. Then the results were used to alter policy. For example, in 2009, we 
created a revised grant application based on our three configuration mod¬ 
els, characteristic implementation features and processes, and the five ef¬ 
fective professional development characteristics (Desimone, 2009). This 
process varied over time as scholarship on STEM partnerships advanced, 
as and we witnessed ITQ partnerships in the field. We discuss this process 
and share examples of how it worked below in the case studies in this spe¬ 
cial issue from three of our ongoing 2007-2011 ITQ projects. 

Next we developed evaluation and research products that we then 
used for member checking. For example, the STPI model was developed 
from the 2005 symposium with input from project partners. Using the STPI 
model, we went to the field and over three years revised STPI in consul¬ 
tation with project directors, evaluators, and other key staff. The October 
2008 symposium used four position papers, developed as our best think¬ 
ing to date about partnerships, to convene ITQ project leaders to: (a) offer 
what we had learned so that projects could learn from each other; (b) to 
elicit responses to improve the papers; and (c) to engage projects in focus 
groups about their collaborations. In 2007, we introduced program theory 
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and logic modeling. Our internal memos were expanded into case studies 
shared with project directors and evaluators for their feedback. Each case 
represented a theory of change as we saw it and was revised using this 
feedback. The projects for their part used the cases to re-imagine and re-ar¬ 
ticulate theories of change in the next granting cycle. This member check¬ 
ing process continues each year at symposia that convene ITQ stakehold¬ 
ers to discuss Center findings, which are then revised to reflect common 
understanding about the status of the ITQ projects. 

Theory Development and Elaboration: 2004-2011 

We began with comparative case studies for meta-evaluation, and 
we have not substantively changed our designs, sampling, or analytic pro¬ 
cesses since 2004. There are currently five team members; the original 
team had four members, three of whom continue. The stability of the team 
translates as continuity and as a viable partnership with IBHE grant ad¬ 
ministrators, with one of these administrators starting in 2004 as well. It 
would be possible to use our current models to evaluate our partnership 
with IBE1E as it has evolved. We began the CSEP/IBE1E partnership with 
a comparative orientation to sharpening policy in successive iterations by 
generalizing findings and applying lessons learned. In this section, we re¬ 
view how our findings shaped policy. Table 1 offers an overview of the 
meta-evaluation, support, and research processes that engaged our team 
from 2004 into the present. All major project events, products, and policy 
changes are indicated. 

Table 1 

ITQ Iterations of Activities, Products, and Policy 


Project year 
# STEM/ 

# all projects Grantwide activities Center products Policy iterations 

2004 Fall and spring None ITQ (No Child Left 

22/26 symposia Behind) initial request 

for proposals (RFP) 
developed from 
fonner Eisenhower 
grants 

Partnerships are 
defined by listing 
potential partners by 
institutional type 
First of annual board 
summaries prepared 
with “lessons learned” 

(continued) 
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Table 1 (continued) 


Project year 
# STEM/ 


# all projects 

Grantwide activities 

Center products 

2005 

Site visits begin 

Proceedings from the 

18/22 

Fall symposium 
focus groups & 
questionnaire for 
project partners 
Results comprise 
Proceedings data 
set 

third symposium: School- 
university partnerships 
for improving teacher 
quality disseminated 
Structures of training and 
processes of implementa¬ 
tion (STPI) model first 
draft developed 

2006 

20/22 

Fall symposium 

STPI model revised 

2007 

Spring symposium 

Collaboration rubric ere- 

6/6 

and bidder’s confer¬ 
ence 

ated from NCATE pro¬ 
fessional development 


schools standards; rubric 
used by grant readers 


2008 

8/10 


Fall symposium 
Position papers 
issued prior to 
symposium 
Rotating focus 
group discussion on 
each research ques¬ 
tion used to refine 
the model 


Four position papers on 
four research questions 
presented 

Draft configurations of 
partnership model 


Policy iterations 


New application 
process 

IBHE commissions 
white paper on col¬ 
laboration from an 
external consultant 
RFP includes over¬ 
view of professional 
development school 
model and includes 
definition of collabo¬ 
ration 

Fewer grants funded 
with expectation that 
grants would be three- 
year and receive full 
funding 

Bidders attend meta¬ 
evaluation seminar 

The IBHE strategic 
plan, Illinois Public 
Agenda for College 
and Career Success, 
approved by trustees 
of the IBHE 
Professional develop¬ 
ment school focus 
evolves to in-service 
orientation; collabora¬ 
tion rubric no longer 
used 


(continued) 
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Table 1 (continued) 


Project year 
# STEM/ 

# all projects 

Grantwide activities 

Center products 

Policy iterations 

2009 

Fall symposium 

Case studies developed 

CSEP meta-evalu- 

9/10 

Center researchers 
share case studies 
and logic models, 
projects respond, 
and cases are 
revised 

for review by grant 
stakeholders 

Stated theories of 
change and logic models 
required for applications 
and renewals 

ation presented to 
the IBHE trustees’ 
monthly meeting in 
April 

Renewal and new 
applications are 
reviewed considering 
theories of change 
Needs assessments 
required for the first 
time 

2010 

Spring and fall 

Revised ITQ RFP based 

New grant award 

6/10 

symposia 

Fall symposium 
included evaluation 
technical support 
meetings 

on meta-evaluation 

process developed 
and piloted 

2011 

Planned activities: 

Planning and Changing 

Increased evaluation 

6/9 

summer evaluation 
webinar 

Fall/spring meta¬ 
evaluation for 
stakeholder com¬ 
munications and 
dissemination 

special issue on school- 
university professional 
development in the 

STEM disciplines 
Summer begins the 
final meta-evaluation 
(2011-2012) 

expectations to link 
teacher and student 
learning 

Meta-analysis to be 
presented to IBHE 
trustees 


Theory-Based Evaluation and ITQ Theories of Change 

We take theory-based evaluation as our overall approach, and all 
ITQ projects operate under three common theories of change. First, each 
ITQ request for proposals (RFP) represents federal and state policy mak¬ 
ers’ current theories about school-university professional development 
partnerships. For example, one implied theory is that STEM profession¬ 
al development requires P-20 partnerships focused on content expertise 
to improve student learning. This theory remains unchanged from 2004. 
Second, we believe that evaluation must be integral to project design and 
develop the capacity among the partners to improve using evaluation re¬ 
sults. Over the years, IBFIE has required detailed evaluation plans that 
bring project-level evaluators to the planning table. This too is a theory 
about the centrality of evaluation. Finally, each P-20 partnership designs 
and executes its own theory of change based on IBFIE’s policy focus and 
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pertinent scholarship. All projects offer activities and interventions they 
believe will yield results. These interventions are causal theories that are 
testable. Logic modeling is used to make theories of change easy to ex¬ 
press and “evaluable.” Here is a simplified version of ITQ’s general theory 
of change (Wholey, et al., 2010, p. 3) represented as a sequence. 

• P-20 Partnership 

• Professional Development 

• Teacher Learning 

• Implementation 

• Student Learning 

Because partnership is critical to each theory of change, our case 
analyses consider partnership structures, processes, and evaluation capacity. 
The sequence above is the basic logic of all ITQ projects, even though “pro¬ 
fessional development” means different things in different projects. Each 
step is a precondition for those that follow: partnerships (and the IBHE’s 
support) are precursors to professional development activities and so on. 
The dependent variable is student achievement, although evaluators contin¬ 
ue to struggle with the methodological issues implied by this causal chain. 
There are two problems creating this situation. First, linking student learn¬ 
ing to professional development is challenging and has been the holy grail 
of professional development evaluation (Guskey, 2000), although No Child 
Left Behind accountability mandates have deepened our knowledge about 
how to do this. But linking partnerships to student learning is more challeng¬ 
ing yet, and our team and each project continues to struggle with a long and 
messy set of potential li nk s asking to be tested. This is a common challenge 
for program theory when programs are complex, and theories are comprised 
of many potential pathways to results (Weiss, 1997). Second, many of these 
projects use constructivist approaches to teaching and learning that are not 
readily captured under current accountability regimes, so each project de¬ 
velops its own evaluation with no specifications from the IBHE beyond 
the program theory orientation. This enables us all to develop new ways 
to capture context-specific outcomes, but it makes comparisons less trans¬ 
parent than if evaluation parameters were specified. In part because of in¬ 
creased grant evaluation expectations, better assessments of teacher and stu¬ 
dent learning (and the connections between them) are developing, but this 
remains challenging and and a probable issue for further research. 

Developmental Stages of ITQ Policy 

The evaluation from 2004 to 2011 has developed recursively as we 
generated two successive structural models for looking at partnerships and 
applied these models to implementation processes and evaluation capacity 
building. In 2004, the first of three evaluation and policy development iter- 
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ations began. Our initial evaluation relied on the professional development 
scholarship available at that time. This was, and still is, a more well-devel¬ 
oped set of theories than those about partnerships, to describe and explain 
what each case might accomplish. One important initial finding was that 
not all 22 of the grant-funded partnerships represented existing, ongoing, 
and authentic collaborations poised to use the grant funds effectively. The 
IBHE recognized that new partnerships needed start-up time for authentic 
collaboration, but the mere availability of funds was a poor beginning. An¬ 
other common oversight was failure to design collaborative, job-embed¬ 
ded professional learning opportunities for teachers. The IBHE grant staff 
realized that professional development regimes not designed with collabo¬ 
ration and evaluation for improvement in mind should: (a) be redesigned 
and (b) be required to evaluate this project element. From that time to this, 
IBHE supports grants to explore ways to use evaluation to draw connec¬ 
tions between teacher and student learning, with implementation as an in¬ 
termediate step. From this beginning, we refocused on partnerships and 
articulated the four core research questions that guide us still. We discuss 
each year from 2004 on considering activities, Center products, and policy 
iterations. We refer the reader to Table 1 for an overview. 

Starting in 2005, we developed structures of training and process¬ 
es of implementation (STPI), a four-part model defining four professional 
development prototypes, from focus groups at the symposium and the first 
rounds of site visits (Baker, Gardner, & Curry, 2008; Gardner, Baker, & Cur¬ 
ry, 2008; Gardner, Pacha, & Baker, 2007; Sappington, Baker, Gardner, & 
Pacha, 2010). Of the four types, only two were considered viable given what 
we knew about effective professional development sufficiently robust to al¬ 
ter student learning and achievement. The two viable models were: (a) on¬ 
going collaborative learning opportunities embedded in school culture and 
work routines and (b) ongoing collaborative learning opportunities shared 
in loosely coupled networks with the university partner as the hub. This lat¬ 
ter model had two specific applications: (a) middle and high school math 
and science teachers who benefitted from regional connections with disci¬ 
pline peers and (b) school and district level administrators convened by a 
regional university for school and district improvement. These networked 
collaborations were facilitated by face-to-face workshops, on-line profes¬ 
sional networks, and shared resources. In 2008-2009, two of ten ITQ part¬ 
nerships used this network model. One appears as a case study in this issue: 
“Improving Science Instruction in Southwestern Illinois and Metro East St. 
Louis: Students Learning Science through a Sustained Network of Teach¬ 
ers,” a partnership project located at Southern Illinois University-Edwards- 
ville (Voss, Khazaeli, Eder, & Gardner, 2011). 

In 2006, the STPI model was revised as we continued our site vis¬ 
its and symposia. Site visits were critical to case development, and each 
site had a primary contact with CSEP and access to a senior researcher pro¬ 
viding expert evaluation support. A new IBHE Executive Director brought 
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new visions for ITQ, which were realized in 2007. Executive leadership 
provides vision, and the new executive envisioned a P-20 system based on 
inter-institutional collaboration. This vision is also the origin of the Illinois 
Public Agenda for College and Career Success, the IBHE (2008) strategic 
plan. A significant policy shift in ITQ resulted: a new emphasis on profes¬ 
sional development (PD) schools as the preferred partnership vehicle. The 
PD school is a well-known model (Holmes Group, 1990; Teitel, 1994). 
The Executive Director commissioned a white paper on collaboration and 
awarded grants favoring the PD school model. The emphasis shifted from 
STEM projects to a more diverse set of educational reforms. In response 
we developed a rubric based on the NCATE Professional Development 
Schools Standard III: Collaboration (National Council for the Accredita¬ 
tion of Teacher Education, 2001). This proved unsatisfactory as the stan¬ 
dard and rubric did not reflect the realities of establishing, developing, 
and sustaining partnerships and only considered PD schools as partnership 
models. Only two of our projects since 2004 have used a PD school mod¬ 
el. Models we found for studying and evaluating collaboration were more 
sanguine about what collaboration could accomplish than our data sug¬ 
gested (Gajda, 2004; Gajda & Koliba, 2007). The models were aspiration- 
al, reflecting the long-standing hopes for productive school and university 
collaboration, but they were not empirically grounded. 

This first year under the new P-20 collaboration policy revealed 
three key findings: (a) there were few authentic PD school model part¬ 
nerships in the state; (b) realizing the model in practice was complex and 
more challenging than policy makers realized; and (c) there were robust 
partnerships doing effective STEM professional development that did not 
match this model. The PD school model faded in subsequent years of re¬ 
fining ITQ policy. One lasting contribution of this policy cycle was a lean¬ 
er set of grants, each funded fully for three years. This policy developed as 
a result of the 2004 evaluation demonstrating that new partnerships need¬ 
ed time to develop capacity. 

Also in 2007, more requirements were made of project-level eval¬ 
uations. Specifically, projects were required to prepare a coherent theory 
of change, using logic modeling as a tool. This was the foundation of proj¬ 
ect case studies in 2008-2009. 

In 2008, the Illinois Public Agenda for College and Career Success 
was accepted by the trustees of the IBHE, and the CSEP team developed and 
shared four position papers at the annual symposium. Each paper addressed 
one of the four research questions. Project partners from around the state had 
advanced access to the papers, and focus groups included all symposium 
participants commenting on all four papers. The results were then shared 
with project directors and evaluators. This work forms the foundation of the 
findings in this special issue. Although we have continued to develop and 
refine our ideas about the structures, implementation processes, and evalua¬ 
tion capacity, we still apply the conceptual framework developed in 2008. 
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Major policy changes came about in 2009 with a completely re¬ 
vised RFP written jointly by CSEP and IB HE partners. This process be¬ 
gan with the development of individual case studies used for comparison. 
The case studies also strengthened support to the projects as site visitors 
used them in discussions with project partnerships along the lines of the 
four research questions. Detailed needs assessments were required for the 
first time, although some projects did them voluntarily right along. By 
now, fewer projects expecting three years of funding were the norm, and 
all continuing projects were required to refine their approaches based on 
statewide evaluation and their own formative and summative evaluations. 
The three case studies in this special issue narrate their change process¬ 
es and evolution of partnership structures, implementation processes, and 
evaluation capacity. 

By 2010, projects that did not support effective models in their 
structures, processes, and evaluation approaches lost funding. A new grant 
award processes and scoring system was initiated that required a theory 
of change based on scholarship and partnership history. A mix of ongoing 
and new projects was funded. We redoubled our efforts with comparative 
cases, now paired to explore contextual variation. (Two case studies pre¬ 
sented here reflect this model. The Southern Illinois University-Carbon- 
dale and University of Chicago partnerships, both elementary math proj¬ 
ects, were paired for data collection and analysis in 2009). The new policy 
completely eliminated projects that used classroom-by-classroom change 
models in favor of those with ongoing collaborative structures and core 
processes (i.e., leadership, expertise, and support). 

The new set of 2010-2011 projects envision STEM professional 
development partnerships in three ways using the three configurations of 
school-university partnerships model to guide their design and develop¬ 
ment. First, the projects share a school core enhancement approach that 
places expertise and support at the disposal of teachers engaged in their 
daily work, although there are variations (Baker et al., 2007; Vogel, 2010). 
Second, all projects are designed for distributed leadership (Cambum, 
Rowan, & Taylor, 2003; Spillane, Diamond, & Jita, 2003). Finally, proj¬ 
ects are designed as networks and rely on resource network supports for 
teachers (Sarason & Lorentz, 1998). 

We move forward in our efforts to understand partnerships that 
face common and specific challenges and build on successes. Now, there 
were fewer projects, nine in all, but all designs use school-based change, 
networked, or combination models, so we can test our theories about their 
uses and potential effectiveness. Now our comparative cases face more 
challenges and successes in common as our recursive learning process in 
evaluation and project synthesis continue. 
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Conclusion 

From 2004 forward, statewide evaluations of school-university 
partnership structures, collaborative implementation processes, and evalu¬ 
ation capacity have been used by researchers and policy makers to make 
judgments about what works. The scholarship on partnerships is descrip¬ 
tive and colored by high hopes about what partnerships can accomplish. 
We have attempted in successive iterations to describe partnerships and 
develop explanations using a comparative case approach. Since 2005, we 
have developed a model that offers three structural views of partnerships 
and we use that lens to characterize collaborative processes that convene 
partners to improve student learning and achievement. We have also tar¬ 
geted evaluation capacity building in our work. Since 2008-2009, we have 
assumed a program theory approach for the full set of grants, individu¬ 
ally and comparatively. The next articles in this special issue of Planning 
and Changing offer a portrait of our best thinking to date, followed by 
three cases studies as representative examples, each representing one of 
the three structural configurations and site-specific approaches to collabo¬ 
ration within that framework. 
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